Add serialize_allocations to chain Slurm allocations - #436

Problem

A workflow whose sequential work outlives any single allocation has no way to keep going. Each allocation runs what fits in its walltime and exits, and nothing brings up the next one. For a 500-job chain at 2-4 hours each — weeks of serial work against a 12-hour partition limit — every existing option is poor:

• torc watch --auto-schedule needs a process alive on the login node for the entire run. Most sites reap long-running login-node processes well before six weeks.
• on_worker_complete + schedule_nodes self-schedules from the compute node, but needs persistent = true to chain more than once — and then never stops. execute_action calls schedule_slurm_nodes unconditionally, so after the last job the exiting worker still submits a successor, which finds no work, exits, and submits another.
• --dependency=singleton via the scheduler's extra gets the flag but not the shared job name that makes it mean anything. Slurm scopes singleton to (user, job name), and torc's generated name embeds the submitting PID and an allocation counter, so no two allocations ever land in the same singleton set.

Change

Adds serialize_allocations to SlurmSchedulerSpec/SlurmSchedulerModel. When set, every allocation for that scheduler submits under one derived name with --dependency=singleton, so Slurm runs them strictly one at a time. Submit N up front and they chain themselves, with nothing running on the login node.

slurm_schedulers:
  - name: chain
    account: my_account
    walltime: "12:00:00"
    serialize_allocations: true

torc slurm schedule-nodes <workflow_id> -n 167
torc slurm create <wf> -n chain -a acct -W 12:00:00 --serialize-allocations
torc slurm update <sched_id> --serialize-allocations true

A queued successor also accrues priority while its predecessor runs — unlike a replacement submitted at shutdown, which enters the queue with no age and pays full wait on every link of the chain.

Design notes

Both halves ride existing extension points, so the diff stays small. The dependency goes through config_map, which already emits #SBATCH --{key}={value}; it is inserted after slurm_defaults so a workflow-level dependency default cannot silently break the chain. The name is decoupled by passing create_submission_script a different first argument than the script path — the unique PID-bearing name stays on the .sh file so concurrent submissions do not overwrite each other. No signature changed, so none of the existing call sites moved.

serialized_slurm_job_name() keys on (workflow_id, scheduler_id) — stable across the separate processes that may submit into one chain, and distinct per scheduler so unrelated allocations do not serialize against each other. The torc- prefix keeps the singleton set clear of the user's other Slurm jobs.

On the scheduler, not the action. Four paths reach schedule_slurm_nodes(): the schedule_nodes action, the CLI, watch --auto-schedule, and recover. Putting the field in action_config would mean someone launching a chain with torc slurm schedule-nodes -n 167 silently gets 167 concurrent allocations. On the scheduler, all four inherit it, so allocations added later join the chain instead of racing it. Auto-generated schedulers (slurm generate/regenerate, scheduler_plan) default to unset — serializing is an opt-in on a user-authored scheduler.

No change needed to allocation cleanup. cancel_unneeded_pending_allocations runs on head-node exit and cancels queued allocations only when no ready/pending/running job remains. The sequential handoff clears that bar comfortably: the background unblock task runs on a 5s interval while an idle worker waits the full 90s before exiting, so the next job is ready long before the check. Surplus allocations are therefore cheap — round the count up and the extras never start.

Testing

• test_serialized_slurm_job_name_is_stable / _scopes_to_workflow_and_scheduler / _honors_job_prefix — the chain only forms if the name never varies with anything process-local.
• test_create_submission_script_with_singleton_dependency — asserts both the --dependency=singleton directive and the shared --job-name land in the script.
• test_slurm_scheduler_serialize_allocations_kdl_roundtrip — this caught a real bug during development: the first emitter wrote a bare true, but the codebase is on KDL v2 (kdl = "6.5") where booleans are #true/#false, so a KDL spec with the field set emitted a file that failed to reparse. Also asserts an omitted value stays None rather than round-tripping to Some(false).

Full suite green (1787 tests), plus cargo fmt --check, clippy --all --all-targets --all-features -D warnings, dprint check, and mdbook build.

Note for reviewers

One thing I could not verify without a live cluster: whether a later duplicate #SBATCH --job-name overrides an earlier one. Nothing in this change depends on it — torc now emits exactly one --job-name, the shared one. It matters only for anyone combining extra: "--job-name=..." with this feature. The docs note that a --dependency set in extra is emitted last and will override the generated one, breaking the chain.

🤖 Generated with Claude Code